【机器翻译】- 神经机器翻译 NMT

Facebook总结的很好

顺便介绍一下CNN缺陷，RNN缺陷

• Convolutional Neural Networks (CNN)
  • Dauphin et al. (2017): Language Modeling with Gated Convolutional Networks
  • Gehring et al. (2017): Convolutional Sequence to Sequence Learning
  • New Edunov et al. (2018): Classical Structured Prediction Losses for Sequence to Sequence Learning
  • New Fan et al. (2018): Hierarchical Neural Story Generation
• Long Short-Term Memory (LSTM) networks
  • Luong et al. (2015): Effective Approaches to Attention-based Neural Machine Translation
  • Wiseman and Rush (2016): Sequence-to-Sequence Learning as Beam-Search Optimization
• Transformer (self-attention) networks
  • Vaswani et al. (2017): Attention Is All You Need
  • New Ott et al. (2018): Scaling Neural Machine Translation

时序建模

见 ml/sequence

其他

通用trick

• gated linear units
• residual connections

模型汇总

模型	paper	出处	简介	创新点	缺陷	code
基于规则的翻译
基于短语的翻译（PBMT）		IBM 1989	较复杂
seq2seq-RNN	Seq2seq Learning with NN	NIPS 2014				code
seq2seq-RNN	Learning Phrase Representations using RNN Encoder-Decoder for SMT	EMNLP2014
☆seq2seq-RNN+attention	Effective Approaches to Attention-based NMT	EMNLP 2015	基于高斯分布推导了Local Attention,比较了Global Align Attention和Local Align Attention， 和视频处理里面 Soft Attention 和 Hard Attention建立了联系。			code
seq2seq-attention	Neural machine translation by jointly learning to align and translate	ICLR 2015	RNN+attention	首次加入attention，ALIGN AND TRANSLATE
	On using very large target vocabulary for neural machine translation	2015	提出词表扩大的方法
☆GNMT	Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation	Google 2016	deep_LSTM, residual, attention, converage，subword
BPE	Neural Machine Translation of Rare Words with Subword Units	2016				code-Sennrich
	Character Based Neural Machine Translation	2016
	Achieving open vocabulary neural machine translation with hybrid word-character models	2016
☆ConvSeq2seq	paper1; paper2	Facebook 2016				code1;
☆Transformer	Attention is all you need	Google 2017				G-tf

扩展阅读

• Google神经网络机器翻译上线历程回顾